{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "81dd54a3-d44a-41ce-aed1-9f3eda29482b",
   "metadata": {},
   "source": [
    "# CI Simulation Repeated (Supplementary Material)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "539ff52f-0cce-4cb9-aff5-91705ee69ad4",
   "metadata": {},
   "source": [
    "- This notebook supplements the [confidence-intervals-for-ml.ipynb](confidence-intervals-for-ml.ipynb) with a case study.\n",
    "\n",
    "- In particular, this notebook repeats the CI simulation from [ci-simulation.ipynb](ci-simulation.ipynb) multiple times to get a sense of which CI method is good at capturing the true model performance.\n",
    "\n",
    "- Just to make sure that there are no silent bugs due to variable naming & reuse, each experiment is in a separate Python script."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a37522e-fa74-4e44-b398-59b7a975d67f",
   "metadata": {},
   "source": [
    "## 1) Normal Approximation Interval Based on the Test Set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b8000642-fb48-499e-828f-24396bdffbec",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "95.6% of 95% confidence intervals contain the true accuracy.\n"
     ]
    }
   ],
   "source": [
    "!python ci-simulation-repeated/1_normal_approx.py --repetitions 1000"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75c876a5-817f-451f-81b0-5af229451e94",
   "metadata": {},
   "source": [
    "## 2.1) A *t* Confidence Interval Interval from Bootstrap Samples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c5540a38-f241-4429-a049-2519291319c0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "98.5% of 95% confidence intervals contain the true accuracy.\n"
     ]
    }
   ],
   "source": [
    "!python ci-simulation-repeated/2.1_bootstrap_t.py --repetitions 1000"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbb37329-4482-4dd4-a6eb-ac76f5ada83c",
   "metadata": {},
   "source": [
    "## 2.2) Bootstrap Percentile Method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "0c6afc91-c960-4959-b7d6-8e212a545281",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "98.0% of 95% confidence intervals contain the true accuracy.\n"
     ]
    }
   ],
   "source": [
    "!python ci-simulation-repeated/2.2_bootstrap_percentile.py --repetitions 1000"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32efa9f6-5190-4726-9c95-ec019dae9960",
   "metadata": {},
   "source": [
    "## 2.3) .632 Bootstrap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4f520295-dbfc-49a7-b8a4-8dea88d6442e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "83.2% of 95% confidence intervals contain the true accuracy.\n"
     ]
    }
   ],
   "source": [
    "!python ci-simulation-repeated/2.3_bootstrap_632.py --repetitions 1000"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eeb6e55a-445d-4f65-b282-85cc2f6a2303",
   "metadata": {},
   "source": [
    "## 3) Bootstrapping the Test Set predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7edb473f-234e-4e4f-b561-45ca5344fcfc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "94.5% of 95% confidence intervals contain the true accuracy.\n"
     ]
    }
   ],
   "source": [
    "!python ci-simulation-repeated/3_bootstrap_test.py --repetitions 1000"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
